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(54) Portable terminal device for transmitting image data via network and image processing 
device for performing an image processing based on recognition result of received image 
data 
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(57) A user of a portable terminal (1) writes sen- 
tences to be transmitted as an e-mail, a mail address, 
and information indicating that service a host device (4) 
is requested to provide is "mail transmission" on paper 
as a memo. By using an image input unit (1a) installed 
in the portable terminal, the paper or the like is imaged. 
The portable terminal transmits the image data thus 
taken in to the host device. The host device analyzes 
received image data by using an image recognition unit 



(5). Upon recognizing by characters that the service 
requested by the portable terminal is "mail transmis- 
sion", the host device starts a mail transmitting/receiv- 
ing unit (6). The mail transmitting/receiving unit (6) 
transmits the content of the written memo included in 
the image data to a terminal specified by the mail 
address included in the image data. 
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Description 

The pfesent invention relates to portable terminal 
device for transmitting image data via network and 
image processing device for performing an image 
processing based on recognition result of received 
image data 

This application is based on Japanese Patent Appli- 
cation No 9-55781 . filed March 1 1 . 1997 and Japanese 
Patent Application No.9-65746, filed March 19. 1997, 
the content of which is incorporated herein by refer- 
ence. 

In recent <*gita!ized society, various documents and 
drawings, especially for business, are handled as an 
electronic tarn In some cases, it is desired to store as 
an electron*; lie even if a content is no more than 
memos. 

In the most general method for generating an elec- 
tronic document, an application program for documen- 
tation is started on a personal computer and a person 
who making a document inputs character by character 
through a keyboard. In the most general method for 
generating electronic drawing, an application program 
for drawing is started and a person who makes draw- 
ings inputs data for the drawing through a keyboard and 
a mouse device. 

Recently, electronic cameras have come into wide 
use. An electronic camera reads image data of an 
object to generate digital data If the electronic camera 
is connected to a personal computer or the like via a 
predetermined interface, the image data read by the 
electronic camera can be processed by that personal 
computer. Therefore, by reading a document written on 
paper with an electronic camera, inputting image data 
read by the electronic camera to a personal computer or 
the like, and executing character recognition process- 
ing, the document written on paper can be converted 
into an electronic file. 

An electronic file is generally formed by using a key- 
board and a mouse. In specific cases, image recogni- 
tion processing is carried out on image data read by 
using an image scanner 

However, it cannot be always said that the manipu- 
lation using the keyboard or mouse is easy for all users. 
There are also users who are not good at manipulation 
of them. Furthermore, in the case where a content of a 
memo is to be converted into an electronic file, it is nec- 
essary to activate an application program for editing 
documents and then start the manipulation each time 
key input is needed. This is troublesome whereas hand- 
writing can be done in a short time. 

On the other hand, in the method of imaging a writ- 
ten content on paper or the like by using an electronic 
camera or the like, it is difficult to input only a necessary 
area and there is a high possibility that a document 
other than a document desired to be subjected to image 
recognition is also input. In this case, it becomes neces- 
sary to display image data read by camera on a per- 



sonal computer display and specify an image 
recognition area, or amending a document obtained as 
an image recognition result. This is also troublesome. 
In the above described method using an electronic 

5 camera, image data obtained at a remote location (such 
as a business trip destination) cannot be converted into 
an electronic file immediately on that site if a user does 
not carry an electronic camera and a notebook compu- 
ter or the like. Furthermore, a character recognition soft- 

10 ware of good performance is in general large in program 
size. It is not desirable to install such a large size pro- 
gram on a simple portable terminal or the like. 

Furthermore, in recent years, an electric mail (e- 
mail) is rapidly coming into wide use for business use 

is and for individual use. The e-mail has a wide range of 
applications ranging from, for example, a small scale 
inter-office mail system for transmitting liaison matters 
in a LAN (local area network) to such a system that an 
e-mail is sent to a desired other party located in all over 

20 the world via the Internet. 

When transmitting an e-mail, usually the user 
inputs a document to be transmitted and a destination 
address by using keys and thus writes a letter. Then the 
user sends out the letter to the network by clicking, for 

25 example, a "transmission" button on an e-mail applica- 
tion. Or in some cases, the document to be transmitted 
is obtained by scanning sentences printed on paper or 
the like as image data by means of an image scanner or 
the like and executing character recognition processing 

30 on the image data, instead of inputting the document to 
be transmitted. Upon receiving a letter (actually a 
packet storing the letter), the network transmits that let- 
ter to the destination specified by the user in accord- 
ance with a destination address set in that letter. 

35 When transmitting an e-mail, usually a character 
string input by the user by means of keys or information 
at least converted to a string of character codes is sent 
out to the network. The destination address is input by 
the user. 

40 There are strong demands for a simple interface in 
use of an e-mail. For example, there are a lot of users 
who desire to input a string of characters by using a 
method other than the key input. In a form known as the 
character input method other than the key input, hand- 

45 writing characters on the display screen of a terminal 
device with a special pen. In some cases, however, it is 
difficult to draw fine patterns. When it is desired to send 
a content already written on paperor the like as it is as 
an e-mail, it becomes necessary to input the content 

so again by using the special pen and this is troublesome. 
Furthermore, there are a lot of scenes in which the user 
sends an e-mail from the movement destination by 
using a portable terminal. Even if a character string writ- 
ten on paper or the like is taken in as image data by 

55 using some means, however, the content thus taken in 
cannot be sent as an e-mail document provided that the 
portable terminal does not have software for character 
recognition. In general, character recognition software 
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of good performance has a large program size. It is not 
desirable to install such program on a simple portable 
terming or the like. 

Accordingly, it is an object of the present invention 
to provide a data transmitting/receiving device capable s 
of transmitting an e-mail having a simple structure and 
with easy operation. 

According to the present invention, there is pro- 
vided a host device for a system connecting various ter- 
minals via a network, comprising means for receiving 10 
data including image data transmitted from a terminal 
and executing character recognition on information of 
transmission destination contained in the image data; 
and means for transmitting information to be transmitted 
included in the received data to a terminal of transmrs- 15 
sion destination obtained by the character recognition. 

According to the present invention, there is pro- 
vided a portable terminal device comprising means for 
reading, as an image, data indicating transmission des- 
tination and transmission data so as to be distinguisha- 20 
We from each other; means for inputting the image data 
thus read and processing instruction for the image data; 
and means for transmitting the image data and the com- 
mand indicating the processing instruction to a host 
device connected to a network 25 

According to the present invention, a document 
including a desired handwritten memo or the like is 
taken in as image data by a camera included in a porta- 
ble terminal, and that handwritten memo or the like can 
be converted to an electronic file by only requesting a 30 
host device to perform recognition processing. In this 
Way, the present invention provides a user interface 
improved in manipulation performance. The image is 
recognized by extracting only a region inside a closed 
loop pattern, a region including an underlined character 35 
string, or a region including a pattern drawn with a spe- 
cific color. As a result, only required information is rec- 
ognized accurately. 

Furthermore, in the present invention, a handwrit- 
ten memo or the like can be simply transmitted to a 40 
desired other party as an e-mail or a fax by taking in a 
document including a desired character string as image 
data with a camera or the like included in a portable ter- 
minal and requesting a necessary service to the host 
device. In this way, the present invention provides a user 45 
interface improved in manipulation performance. 

Furthermore, since image data processing required 
to be performed at high speed, especially pattern recog- 
nition processing and character recognition processing 
are not executed on the side of the portable terminal, it $0 
is possible to obtain a result of sophisticated image data 
processing without providing a large capacity memory 
or a high performance processor. 

The invention can be more fully understood from 
the following detailed description when taken in con- 55 
junction with the accompanying drawings, in which; 

FIG. 1 is a system configuration diagram of an 



embodiment of the present invention; 
FIG. 2 is an exterior view of a portable terminal; 
FIG. 3 is a configuration diagram of the portable ter- 
minal; 

FIG. 4 is a flow chart of processing of transmitting 
image data taken in by imaging with a camera in the 
potable terminal to a host device and requesting a 
predetermined service; 

FIGS. 5A and 5B show first examples of image data 
input by a camera and displayed on a display unit of 
the portable terminal; 

FIGS. 6A and 6B show second examples of image 
data input by a camera and displayed on the display 
unit of the portable terminal; 
FIG. 7 shows third examples of image data input by 
a camera and displayed on the display unit of the 
portable terminal; 

FIGS. 8A and 8B show fourth examples of image 
data input by a camera and displayed on the display 
unit of the portable terminal; 
FIG. 9 is a diagram illustrating the concept of "rec- 
ognition'' of the present embodiment; 
FIG. 10 A is a diagram showing the structure of a 
. packet send out from the portable terminal; 
FIG. 1 0B is a detailed flow chart of packet creation 
processing; 

FIG. 1 1 is a flow chart of processing for extracting 
image data of a recognition subject; 
FIG. 12 is a diagram showing an example of appli- 
cation of image processing shown in FIG. 4; 
FIGS. 13A and 13B show examples of discrimina- 
tion of an image to be recognized; 
FIGS. 14A and 14B is a flow chart illustrating 
processing tor extracting image data of a recogni- 
tion subject in the case where a memo or the like is 
written as shown in FIGS. 13A and 13B; 
FIG. 15 is a configuration diagram of a host device; 
FIGS. 16 and 17 are flow charts illustrating 
processing of the host device; 
FIG. 18 is a flow chart illustrating processing for 
recognizing service requested by the user from 
received image data; 

FIG. 19 is a flow chart illustrating processing for 
extracting a character string of mail sentences from 
received image data; 

FIG. 20 is a flow chart illustrating mail transmission 
processing; 

FIG. 21 shows a diagramlilustrating a packet stor- 
ing mail sentences; 

FIG. 22 is a flow chart illustrating processing per- 
formed when a mail inquiry is received from another 
terminal; and 

FIG. 23 is a configuration diagram of a table storing 
e-mail addresses belonging to each group so as to 
be associated with a group name. 

A preferred embodiment of a data transmit- 
ting/receiving device according to the present invention 



EP 0 865 192 A2 



wiD now be described with reference to the accompany- 
ing drawings. 

FIG. Vfis a system configuration diagram of a first 
embodiment of the present invention. A portable termi- 
nal 1 has a communication function, and is capable of 
requesting another device connected to a network 3 to 
perform given data processing, receiving a result 
thereof, and displaying it The portable terminal 1 is con- 
nected to the network 3 by using a scheme of transmit- 
ting/receiving data on a radio channel via a nearby base 
station, a scheme of transmitting/receiving data to/from 
a darice having a communication function (optical link 
unit) by using Ir (infrared ray) communication or the like, 
or a wired scheme of transmitting/receiving data. The 
portable terminal 1 has an image input unit 1a including 
an electronic camera. The portable terminal 1 has a 
function of sending image data taken in by using the 
image input unit 1a to another device (such as host 
device 4) connected to the network 3 and requesting the 
other device to perform image processing. Further- 
more, the portable terminal 1 is also capable of process- 
ing the image data taken in by using the image input unit 
1a, and displays image data taken in or processed on 
an image display unit 1c. 

The network 3 is a public telephone network, a PHS 
network, or a LAN. The host device 4 is accommodated 
in the network 3. The host device 4 is a server machine. 
In accordance with a request transferred from the port- 
able terminal 1, the host device 4 executes data 
processing. The host device 4 has an image recognition 
unit 5, a mail transmitting/receiving unit 6, and a fax 
transmitting/receiving unit 7. The image recognition unit 
5 executes pattern recognition processing or character 
recognition processing on image data transmitted from 
the portable terminal 1, and detects a predetermined 
pattern and a predetermined character string contained 
in the received image data. The mail transmit- 
ting/receiving unit 6 has a function of transmit- 
ting/receiving an e-mail to/from another terminal 9 
connected to via, for example Internet 8, and is capable 
of transmitting character string data recognized by the 
image recognition unit 6 to the other terminal 9. The fax 
transmitting/receiving unit 7 has a function of transmit- 
ting/receiving a fax signal to/from a fax terminal 10 con- 
nected via, for example, the network 3, and is capable of 
transmitting image data sent from the portable terminal 
1 to the fax terminal 1 0. 

When the user of the portable terminal 1 wishes to 
transmit a content written on, for example, paper or the 
like as a memo to the terminal 9 as an e-mail, the user 
adds the mail address of the terminal 9 and information 
indicating that service which is requested to the host 
device 4 is "mail transmission" to the written memo as 
image information on the paper, takes in the image on 
the paper by using the image input unit 2 (the electric 
camera connected to the portable terminal 1), and 
transmits the image data thus taken in to the host device 
4. The host device 4 analyzes the received image data 



by using the image recognition unit 5. Upon recognizing 
that the service which is requested to the host device 4 
from the portable terminal 1 is "mail transmission," the 
host device 4 starts the mail transmitting/receiving unit 

5 6, and transmits the content of the written memo con- 
tained in the image data to the mail address contained 
in the image data. 

In the same way. when the user of the portable ter- 
minal 1 wishes to transmit the content written on, for 

10 example, paper or the like as a memo to the fax terminal 
10 as a fax transmission, the user adds the telephone 
number of the fax terminal 1 0 and information indicating 
that service the host device 4 is requested to provide is 
"fax transmission" to the written memo as image infor- 
ms mation on the paper, takes in the image on the paper by 
using the image unit 2, and transmits the image data 
thus taken in to the host device 4. Upon recognizing that 
the service requested by the portable terminal 1 is "fax 
transmission," the host device 4 starts the fax transmit- 

20 ting/receiving unit 7. and transmits the content of the 
written memo contained in the image data to a device 
having the telephone number contained in the image 
data, as a fax transmission. 

The host device 4 may be formed by a single com- 

25 puter. Or an image processing server device, a mail 
server device, and a fax server device may be formed 
by respectively independent computers. The host 
device 4 is a server computer provided at an Internet 
provider, a personal computer communication com- 

30 pany, or an Intranet managing section. 

When it is desired to perform an e-mail transmis- 
sion or a fax transmission in the system of the present 
embodiment it is necessary only to scan a content writ- 
ten on the document desired to be transmitted by using 

35 a camera or the like and transmit the scan data to the 
host device 4 as image data. Therefore, the user can 
receive the e-mail transmission service or fax transmis- 
sion service by only writing desired characters and/or 
figures on paper or the like as a memo and reading it 

40 with the camera. Furthermore, the portable terminal 1 
needs only to have the function of taking in image data 
and the communication function as a constituent condi- 
tion for receiving the above described service. As a 
result, a reduced weight and a lowered cost can be 

45 accomplished. 

When the user of the portable terminal 1 wishes to 
convert a memo handwritten on, for example, paper or 
the like into an electric file, the user draws a closed loop 
pattern so as to surround a region on paper in which the 

so memo or the like is written, and the image on paper is 
read by using the image input unit 1a. The image 
processing unit 1b analyzes the image data read by the 
input unit 1a. Upon detecting a closed loop pattern, the 
image processing unit 1b extracts image data of a 

55 region inside the closed loop pattern. In the extracted 
image data, unnecessary regions have been deleted 
from the memo or the like written on paper. The portable 
terminal 1 transmits image pattern data extracted by the 
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image processing unit 1b to the host device 4 via the 
network 3. 

U&on receiving image pattern data from the porta- 
ble terminal 1 , the host device 4 causes the image rec- 
ognition unit 5 to analyze the image pattern. Here the 
image pattern analyzed and recognized by the image 
recognition unit 5 is the content itseH written on paper or 
the like by the user, and an unnecessary image is 
removed. Therefore, the host device 4 can obtain only 
the content written on paper or the like by the user as a 
result of recognition. 

Alternatively, such a configuration that the process- 
ing of the image processing unit 1b is executed by the 
host device 4 may be adopted. In this case, the portable 
terminal 1 transmits the image read by using the image 
input unit 1a as it is, and the image recognition unit 5 
extracts image data located in a region inside the closed 
loop pattern, and analyzes the extracted image pattern. 

The result of recognition performed by the image 
recognition unit 5 may be stored in the host device 4, or 
it may be transmitted to the portable terminal 1 automat- 
ically or in response to demands. In this case, the port- 
able terminal 1 displays the recognition result received 
from the host device 4 on the image display unit 1 c. Fur- 
thermore, the result of recognition performed by the rec- 
ognition unit 5 may be transmitted to a predetermined 
terminal device. Furthermore, retrieval processing may 
be executed by using the recognition result as a key- 
word, and the recognition result may be translated to a 
predetermined language. 

The image recognition unit 5 may be implemented 
as a function obtained by execution of a predetermined 
program by the host device 4, or may be implemented 
by a computer which is independent of the host device 
4. 

When it is desired to convert characters or figures 
written on paper or the like as a memo into an electric 
file (digital data) in the system of the present embodi- 
ment, the user needs only draw a closed loop pattern so 
as to surround a region on paper in which the memo or 
the like is written, image it with a camera or the like, and 
transmit the image data (or image data subjected to pre- 
determined image processing) to the host device 4. In 
other words, it is only required for a user to surround the 
memo or the like with a closed loop pattern and imaging 
it with acamera or the like. It is very simple. As the con- 
stituent condition for recognizing the above described 
memo a the like, the portable terminal 1 need not have 
the character recognition function, and needs only have 
the function to read an image and the communication 
function. Depending on the configuration, the function of 
detecting a closed loop pattern is also required. There- 
fore, the weight ami cost can be reduced. 

FIG. 2 is an exterior view of the portable terminal 1 . 
The portable terminal 1 has an LCD display unit 11. a 
camera 12, an IC card slot 13 and a radio communica- 
tion antenna 14, Furthermore, in order to support other 
communication schemes, the portable terminal 1 has 



an Ir communication socket 15 and a wire communica- 
tion socket 16. The LCD display unit 11 has a touch 
panel function. Therefore, the portable terminal 1 can 
process handwritten input information. Numeral 17 
5 denotes a power switch. 

FIG. 3 is a configuration diagram of the portable ter- 
minal 1. A CPU 21 executes a program stored in a stor- 
age device (a ROM and a RAM) 22. The CPU 21 and 
the storage device 22 are interconnected via a bus 23. 
10 The storage device 22 is formed by a semiconduc- 
tor memory, a magnetic recording medium, or an optical 
recording medium, and stores programs and data. The 
storage device 22 may be fixedly mounted in the porta- 
ble terminal 1 , or it may be detachably mounted. 
15 A recording medium driver 24 is connected to the 
bus 23. The recording medium driver 24 is a device for 
reading data stored in a dechatable recording medium 
25 (including a semiconductor memory, a magnetic 
disk, an optical disk, or an optical magnetic disk) or writ- 
20 ing data into the detachable recording medium 25. It is 
assumed that an IC card is used as an example of the 
detachable recording medium 25. The CPU 21 can also 
execute a program stored in the detachable recording 
rxiedium 25. 

25 Programs and data recorded in the storage device 
22 may be received from another device connected via 
a communication line or the like and recorded therein. 
Furthermore, the CPU 21 may use a program and data 
stored in a storage device installed on the side of 
30 another device via a communication line or the like. 

A unit corresponding to the LCD display unit 1 1 is 
formed of a liquid crystal display (LCD) 31, a memory 
32 for storing information to be displayed on the liquid 
crystal display (LCD) 31 , an LCD driver 33 for outputtmg 
35 information stored in the memory 32 to the liquid crystal 
display 31 under the control of the LCD controller 34, an 
LCD controller 34 for controlling the memory 32 and the 
LCD driver 33, a touch panel unit 35, and a touch panel 
controller 36 informing the CPU 21 of input information 
40 detected by the touch panel unit 35. 

The camera 12 is, for example, a CCD camera. 
Output of the CCD camera 12 is converted to digital 
data of bit map form by an A/D converter 37, and written 
into a memory 38. A camera controller 39 outputs 
45 image data stored in the memory 38 to the bus 23 
based on a command given by the CPU 21 . 

When sending out data from the portable terminal 
1, a communication controller^ generates a transmis- 
sion packet based on a command given by the CPU 21 
so and transmits it to a radio transceiver 41 or 42, or a wire 
transceiver 43. When receiving data, the communica- 
tion controller 40 outputs data stored in a packet 
received via the radio transceiver 41 or 42, or the wire 
transceiver 43 onto the bus 23. The radio transceiver 41 
55 is connected to the radio communication antenna 14 
shown in FIG. 2, and transmits/receives radio data 
to/from a radio base station. The radio transceiver 42 is 
a transceiver for performing Ir communication. The wire 
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transceiver 43 is, lor example, a modem. The radio 
transceiver 42 and the wire transceiver 43 are optionally 
mounted, trhe portable terminal 1 furthermore has a 
clock 44. 

FIG. 4 is a flow chart showing processing of trans- 
mitting image data taken in by the camera 12 of the 
portable terminal 1 to the host device 4 and requesting 
desired service. Programs implementing respective 
functions shown in this flow chert are stored in the stor- 
I age device 22 in a form of program code which can be 

! read by the CPU 21. 

This flow chart shows processing performed after 
1 the portable terminal 1 detects some input in an input 

| wait state. Furthermore, in this embodiment, an exam- 

ple in which characters or the like written by the user as 
a memo are input by a camera will now be described. 
| , At step Si , rt is determined whether the detected 

i input is an input from the camera 1 2, processing of step 

| S2 and subsequent steps is executed. If it is an input 

from different one, other processing corresponding to 
that input is executed at step S13. If the characters or 
the like are input by the camera 12 by the user, process- 
ing of step S2 and subsequent steps is started, ff there 
i is the camera input, time information output from the 

| clock 44 is stored. 

At step S2, image data taken in by the camera 12 is 
displayed on the LCD display unit 11. In other words, 
image data taken in by the camera 1 2 is written into the 
memory 32 of the LCD display unit 11, and the image 
data is displayed on the liquid crystal display 31. As a 
result, the user can see the input image on the spot. 
FIG. 5A shows an example of display of image data 
taken in by the camera. In this example, the user writes 
j mail sentences, "Hello! Thank you for your e-mail. As for 

the schedule of the next week, ..." as a memo. Further- 
more, "E-mail transmission" is written as service of a 
request made to the host device 4 and "abc@xyz.co.jp" 
is written as address of mail transmission destination. A 
camera input range Q having these messages written 
therein is imaged. Each of the specified sen/ice and the 
address is put in brackets. 

For transmitting image data to the host device 4 and 
causing the host device 4 to perform data transmission 
processing in accordance with information contained in 
that image data, the information specifying service and 
information specifying the mail transmission destination 
must be recognizable with respect to the mail sentences 
(mail content) as described before with reference to 
FIG. 1 . In the example of FIG. 5A, it was made a rule to 
put each of the information specifying the service and 
j the information specifying the mail transmission desti- 

> nation in brackets. The brackets may be replaced with 

| other punctuation, such as parentheses. Alternatively, it 

| may be made a rule to effect character decoration such 

i as underlines on the information specifying the service 

! and the information specifying the mail transmission 

destination. Furthermore, the service classification may 
be associated with a color. For example, it is determined 



beforehand that a command indicating the mail trans- 
mission service is represented by a red color and a 
command indicating the fax transmission service is rep- 
resented by a blue color. It is made a rule that a content 

5 written with such a color specifies the transmission des- 
tination. In an example shown in FIG. 5B, 
"abc@xyz.co.jp" is written with the red color. By analyz- 
ing image data containing this image, therefore, the host 
device 4 can recognize the service requested by the 

w user as the "E-mail transmission" and the transmission 
destination address as the "abc@xy2.c0.jp." In any 
event, the command indicating the service and the infor- 
mation specifying the mail transmission destination are 
distinguished by discrimination information such as the 

is symbols, character decoration, or color. 

When it is desired to transmit an image input by the 
camera 12 to a desired fax terminal as a fax transmis- 
sion, the user writes "Fax transmission** near a memo 
(here, a map) desired to be transmitted as the 

20 requested service for the host device 4, and puts the 
character string in brackets, as shown in FIG. 6A. Fur- 
thermore, the user writes the telephone number of the 
fax terminal of the transmission destination as well 
together with brackets. 

25 When reading the content to be transmitted as an 
e-mail or fax transmission by using the camera 12, a 
predetermined format may be used. FIG. 6B shows an 
example of a format for entering the content desired to 
be transmitted as an e-mail. The format has an address 

30 entering region and a mail sentence entering region. 

As for the service that the host device 4 is 
requested to provide, a received mail inquiry is also 
possible besides transmission of data to a desired 
device such as an e-mail transmission or a fax transmis- 

35 sion. In the received mail inquiry, it is requested to 
download a mail addressed to the user which is arrived 
at a mail server (the mail transmitting/receiving unit 6 of 
FIG. 1) to the portable terminal 1. In this case, the user 
writes "Mail inquiry" on paper or the like as a memo of a 

40 request command for the host device 4, and puts the 
character string in brackets as shown in FIG. 7. When 
the user wishes to read only a mail coming from a par- 
ticular person, the user writes, for example, "from Mr. K" 
on paper. The camera inputting range Q of the paper 

45 having the written memo thereon is imaged by the cam- 
era 12, and the input image data is transmitted to the 
host device 4. 

Returning to the flow chart of FIG. 4, it is deter- 
mined at step S3 whether the camera input image data 

so is amended. Here, an example of manipulation of the 
user performed when a screen shown in FIG. 8A is dis- 
played by the processing of step S2 will now be 
described. When the user wishes to amend the content 
of the displayed screen, the user inputs the amending 

55 content on the LCD display unit 1 1 by using a handwrit- 
ing pen. For example, when "By the way," within "A port- 
able terminal of xxxx put on the market this month has 
enriched mail functions By the way, ... is unnec- 
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essary, a deletion symbol is written on "By the way,* by 
using the handwriting pen on the LCD display unit 1 1 as 
showrt in FIG. 8B. When the amendment of the dis- 
played content is finished, the user inputs the amend- 
ment completion. This processing is performed by, for 5 
example, displaying an "amendment end" button as well 
when displaying the image input by the camera 12 on 
the LCD display unit 11. The user presses that button 
with the handwriting pen. In the case where there is no 
amendment the user presses the amendment end but- w 
ton without amending the image input by the camera 12. 

In keeping with the above described operation of 
the user, steps S3 and S4 are executed. In other words, 
if it is detected at step S3 that the amendment end but- 
ton is pressed, processing advances to step S4. When is 
the user presses the amendment end button without 
amending image data displayed on the LCD display unit 
1 1 , step S4 is skipped and the processing proceeds to 
step S5. 

At step S4, the image data is updated according to 20 
pen input from the user. In other wads, a locus pattern 
pressed on the LCD display unit 1 1 by the pen input is 
overwritten in the memory 32 as data of a bit map form. 
At this time, the image data currently displayed on the 
LCD display unit 1 1 is stored in the memory 32. There- 25 
fore, the image data taken in by camera and the locus 
pattern written by pen input are merged. 

At step S5, a screen for causing the user to specify 
whether the image taken in by camera should be trans- 
mitted to the host device 4 is displayed on the LCD dis- 30 
play unit 1 1 . At this time, the image data stored in the 
memory 32 is saved in the RAM of the storage device 
22. Image data of the screen for causing the user to 
specify desired service is read out from the ROM of the 
storage device 22 and written into the memory 32. The 35 
user's command input screen includes buttons for input- 
ting processing commands, such as "transmission", 
"recognition", "storage", and "cancel". 

If the user inputs processing command on the 
user's command input screen, i.e., the user presses one 40 
of displayed buttons, then processing of step S6 and the 
subsequent steps is executed. First, the user's com- 
mand is recognized at step S6. At step S7, it is deter- 
mined whether the user's command is "transmission". If 
the transmission button is pressed, the user's command 45 
is judged to include a command such as an e-mail 
transmission or a fax transmission and the processing 
proceeds to step S8. 

At step S8, a packet for storing image data when 
transmitting the image data taken in by camera to the so 
host device 4 is created. The packet creation process- 
ing will be described later. At step S9, the packet cre- 
ated at step S8 is sent out to the network 3 shown in 
FIG. 1. 

If the user commands "recognition" on the user's ss 
command input screen, then it is detected at step S10 
that the user presses the recognition button, the user's 
command is determined to be image recognition serv- 



ice, and the processing proceeds to step S1 1 . 

FIG. 9 is a diagram illustrating the concept of the 
image recognition service of the present embodiment, 
and it shows an example of an image obtained when a 
memo handwritten on paper by the user is imaged by 
the camera. In this example, a memo is written as 
"Meeting with Mr. K on August 18" in a blank space of 
paper having alphabets A through Z previously written 
thereon, and that memo is read by the camera 12. In 
order to discriminate between a portion desired to be 
subjected to character recognition and remaining por- 
tions, the user draws such a closed loop as to surround 
the memo sentence desired to be subjected to charac- 
ter recognition on paper. This closed loop may be drawn 
on paper by using an ordinary pen or pencil. 

The user scans the paper so that at least the memo 
sentence desired to be subjected to character recogni- 
tion and the closed loop pattern surrounding the memo 
sentence may be read. At this time in the example 
shown in FIG. 9, unnecessary images located in a cam- 
era inputting range P, i.e., alphabetic letters Q through Z 
are read as image data together with the memo. 

Even in this state, the image recognition processing 
is executed, in the image processing of the present 
embodiment, with respect to a region "a" located inside 
the closed loop pattern. Therefore, alphabetic letters Q 
through Z located outside the closed loop pattern are 
not recognized, but only the memo written as "Meeting 
with Mr. K on August 18" is recognized. In this way, only 
the desired pattern can be recognized while deleting 
unnecessary patterns with a user interface. 

Returning to the f bw chart of FIG. 4 again, image 
data input by the camera 12 is temporarily stored in a 
RAM region of the storage device 22, and image data of 
the recognition subject is extracted from the stored 
image data at step S11. In the example shown in, for 
example, FIG. 9, the region located inside the closed 
loop pattern is regarded as the region of the recognition 
subject, and the image data of that region is extracted. 
The processing of step S1 1 will be described later in 
detail. Subsequently, at step S8, the packet for transmit- 
ting the image data extracted at step S7 to the host unit 
4 is created. The processing of step S8 will also be 
described later. At step S9, the packet created at step 
S8 is sent out to the network 3 shown in FIG. 1 . 

On the other hand, if the user's command is judged 
at step S10 not to be "recognition," it is determined at 
step S12 whether the user presses the storage button. 
If the user presses the storage button, image data is 
stored on the side of the portable terminal 1 at step S14. 
On the other hand, if the user does not press the stor- 
age button, it is considered that the user presses the 
cancel button and the image data is discarded at step 
S13. 

Upon reading the image in which the region of the 
recognition subject is represented by a substantially 
closed loop pattern, the portable terminal 1 regards only 
the region inside the substantial closed loop pattern as 
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the recpon of the recognition subject, extracts the 
region, and sends the extracted image data of the 
region of ttfe recognition subject to the host device 4, As 
described above, the extracted image data contains the 
region desired to be actually subjected to image recog- 
nition, and an unnecessary image is deleted therefrom. 

FIG. 1 0A is a diagram showing the structure of a 
packet sent out from the portable terminal 1. This 
packet is created at step S8 shown in FIG. 4. 

Each packet is formed of a header portion and a 
data portion. The header portion stores a transmission 
source address, a destination address, arc! the like. 
Which system addresses stored as the transmission 
source address and the destination address belong to 
depends upon the network configuration to which the 
present embodiment is applied. In the TCP/IP commu- 
nication, for example. IP addresses are stored. 

In the data portion, application identification infor- 
mation, terminal ID. image data, time information, and 
the like are stored. The application identification infor- 
mation is information identifying an application program 
to be started in the destination terminal (here, the host 
device 4). In other words, the application identification 
information is a command for starting a predetermined 
program in the terminal of the reception destination. In 
the present embodiment, a command identifying the 
mail transmission, fax transmission, mail readout, fax 
readout, recognition, or the like is set. In the TCP/IP 
communication, the application identification informa- 
tion is specified as a port number. 

The terminal ID is an identification number of the 
portable terminal 1 . The image data is image data taken 
in by the camera, and is compressed when stored in a 
packet. H there is an input with the handwriting pen, the 
locus pattern of the pen input is also merged as image 
data as described before with reference to FIG. 8. The 
time information is information representing the time 
when the image data sent out to the host device 4 was 
irput by the camera 

FIG. 1 0B is a detailed flow chart of the packet crea- 
tion processing of step S8 shown in FIG. 4. At step S21 ( 
the image data to be transmitted to the host device 4 is 
compressed and stored in the data portion. The com- 
pression scheme is, for example, the JPEG. At step 
S22, the time information representing the time when 
the above described image data is input by the camera 
is set. At step S23. information identifying an image 
data analysis and recognition program is set as the 
application identification information. At step S24, a ter- 
minal ID is set as the information identifying the portable 
terminal 1 (i.a, information identifying its own device). 
Furthermore, at step S25, the header portion is created. 
In the header portion, at least the address of the porta- 
ble terminal 1 (i.e., the address of its own device) arid 
the address of the host device 4 are set as the transmis- 
sion source address and the destination address, 
respectively. 

The packet created as described above is sent out 



to the network 3. In accordance with the destination 
address of the packet, the network 3 transmits the 
packet to the host device 4. 

FIG. 1 1 is a detailed flow chart of the processing of 
s step S1 1 shown in FIG. 4, and it shows processing for 
extracting the image data of the recognition subject. At 
step S31 , the image data read by the camera 1 2 is read 
out from the RAM region of the storage device 22 and 
subjected to pre-processing in order to remove noise of 
10 the image data, and thereafter line detection processing 
is executed. The line detection processing is a known 
technique. For example, when density distribution or 
color distribution of the image data is examined, a line" 
can be detected by connecting points each having a 
is large change ratio. Also in a boundary between a cer- 
tain region and another region, the change ratio of the 
density distribution or color distribution becomes large. 
However, discrimination between a line* and a "bound- 
ary between regions" is performed in a conventional 
20 technique. By thus examining the density distribution or 
color distribution in the entire region of the image data 
subjected to the above described pre-processing, the 
"line" contained in the image is detected. 

At step S32. line connection processing for con- 
25 necting a break of the "line" detected at step S31 is exe- 
cuted. For example, in the example shown in FIG. 9, the 
closed loop pattern substantially surrounds the hand- 
written memo. However, a part thereof is broken, and 
there is so-called "line break." The line connection 
30 processing is processing for repairing such a "line 
break." This is also a known technique. For example, 
from a tip of a line on which the line break" occurs, a 
segment of a line having a predetermined length is 
extended in the extension direction of the line. Around 
35 the tip of the line, the extended segment of the line is 
revolved in a predetermined angle range. If the segment 
of the line comes in contact with another line, process- 
ing for connecting those two lines is performed. By this 
line connection processing, the closed loop pattern 
40 interrupted in part (substantially closed loop pattern) is 
repaired to become a completely closed loop pattern. 

Among closed loop patterns contained in the image 
data, such a closed loop pattern that the area of the 
region sunounded thereby is greater than a predeter- 
45 mined value is detected at step S33. For example, there 
is a possibility that an alphabetic letter "Q" in FIG. 9 is 
regatded as a closed loop pattern. By executing step 
S33, therefore, closed loop patterns other than such a 
closed loop pattern for specifying the region of the rec- 
so ognition subject are excluded. The number of closed 
loop patterns detected by this processing need not be 
one. but may be two or more. At step S34, the image 
data in the region surrounded by the closed loop pattern 
detected at step S33 is output. 
55 FIG. 1 2 is a diagram showing an example of appli- 
cation of the above described image processing. As 
described above, a boundary between regions can be 
detected by examining the density distribution or color 
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distribution of the image data. In the example shown in 
FIG. 12, this technique is used. It is now assumed that 
a memb desired to be subjected to the image recogni- 
tion is written on paper Q. The paper Q having the 
memo written thereon is placed on a desk or the like s 
having a color different from that of the paper Q. When 
imaging the paper Q by using the camera 12, a camera 
input region P including the paper Q is imaged. 

For the image data within the camera input region P 
thus taken in, step S1 1 of FIG. 4 is executed. Since the 10 
color of the paper is different from the color of the top 
surface of the desk, the peripheral edge of the paper is 
detected as a closed loop pattern, and only the image of 
the paper is extracted as the image data of the recogni- 
tion subject. If the imaging method is thus changed, the is 
necessary region can be extracted without drawing a 
closed loop pattern surrounding the memo on paper 
with a pen or a pencil. 

In the example shown in FIG. 9 or 12, the region of 
the image recognition subject is distinguished from an 20 
unnecessary region by surrounding the handwritten 
memo with a substantially closed loop pattern. How- 
ever, a different method may also be used. For example, 
it may be made a rule to "underline a character string to 
be recognized" and only the underlined character string 25 
may be extracted from the image data taken in by using 
the camera 12. In the example shown in FIG. 13 A, only 
"Meeting with Mr. K on August 18" is underlined. In this 
case, alphabetic letters Q through Z are not extracted, 
and only "Meeting with Mr. K on August 1 8" is detected. 30 

FIG. 14A is a flow chart illustrating the recognition 
subject image data extraction processing in the case 
where the character string to be recognized is identified 
by underlining. In the flow chart of FIG. 4, this process- 
ing is executed as step S1 1 . 35 

Steps S41 and S42 are the same as steps S31 and 
S32 of FIG. 11, respectively. "Lines" are detected from 
the image data read by the camera 12, and the "lines" 
are repaired. Among the "lines" obtained by the above 
described processing, "lines" longer than a predeter- 40 
mined length in the horizontal direction or a nearly hori- 
zontal direction are detected at step S43. The "lines" 
detected here may be plural. In the example shown in 
FIG. 13 A, two lines are detected. Along each of "lines" 
detected at step S43, a region above the "line" is cut out 45 
with a predetermined width, and the cut out regions are 
output as recognition subject regions at step S44. 

As a method for identifying the subject region of the 
image recognition, a color may also be used. For exam- 
ple, H is made a rule to "write the character string with a so 
red cola." Among the image data taken in by using the 
camera 12, only the character string written with a red 
color is extracted. In the example shown in FIG. 136, 
only the characters of "Meeting with Mr. K on August 1 8" 
are written with a red color, and remaining characters ss 
are written with a black color. In this case, the alphabetic 
letters Q through Z are not extracted, and only "Meeting 
with Mr. K on August 1 8" is extracted. 



FIG. 14B is a flow chart illustrating the extraction 
processing of recognition subject image data in the 
case where the character string of the recognition sub- 
ject is written with a color specified beforehand for iden- 
tification. This processing is also executed as step S7 in 
the flow chart of FIG. 4. At step S51 , a region containing 
pixels of a color specified beforehand with a predeter- 
mined density or more is cut out from the image data 
taken in by using the camera 12, and image data 
included in the cut out region is output as the recogni- 
tion subject region. The character string of the recogni- 
tion subject may be specified by using other character 
decoration information instead of the underline or color. 

FIG. 15 is a configuration diagram of the host 
device 4. A storage device 51 is formed by a semicon- 
ductor memory, a magnetic recording medium, or an 
optical recording medium, and stores programs, data 
and the like. The storage device 51 may be fixedly 
mounted in the host device 4, or may be mounted so as 
to be freely attached and detached. 

A recording medium driver 52 is a device for read- 
ing out data stored in a detachable recording medium 
(including a semiconductor memory, a magnetic disk, 
an optical disk, an optical magnetic disk, or the like) 53, 
or writing data into the detachable recording medium 
53. A communication controller 54 is a unit for control- 
ling exchange of data with respect to the network. 
Transmission and reception of packets with respect to 
the portable terminal 1 are also controlled by the com- 
munication controller 54. 

A CPU 55 loads a program or the like from the stor- 
age device 51 or the detachable recording medium 53 
into a memory 56, and executes it. The program, data, 
and the like recorded in the storage device 51 may be 
obtained by writing the program, data, and the like 
stored in the detachable recording medium 53 into the 
storage device 51, or receiving the program, data, and 
the like from another device on the network via a com- 
munication line or the like and recording it into the stor- 
age device 51. Furthermore, the CPU 55 may use the 
program, data, and the like stored in another device 
installed on the network via a communication line or the 
like. The image recognition unit 5 ( the mail transmit- 
ting/receiving unit 6, and the fax transmitting/receiving 
unit 7 shown in FIG. 1 represent the functions executed 
by the CPU 55 and the program via the communication 
controller 54 as blocks. 

FIGS. 16 and 17 are TTow charts illustrating 
processing of the host device 4. The host device 4 
receives a packet (i.e., a packet created by the process- 
ing of the flow chart of FIG. 4) sent out from the portable 
terminal 1, and processes the image data in accord- 
ance with information set in its data portion. The opera- 
tion performed when the host device 4 has received a 
packet from the network 3 will now be described. 

Upon receiving a packet at step $31, the terminal 
which sent out the packet is recognized by examining 
the terminal ID set in the packet at step S32. At step 
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S33, it is determined whether the received packet con- 
tains e-mail data, If e-mail data is contained, then it is 
recognized 5 ! as mail receiving and the processing pro- 
ceeds to step S41. At step S41, the mail transmit- 
ting/receiving unit 6 is started and requested to perform 
processing concerning mail receiving. The mail trans- 
mitting/receiving unit 6 stores the received e-mail by 
using its destination as a key, and informs the destina- 
tion user the reception of the mail in response to 
demands. 

At step S34, it is determined whether the received 
packet contains fax data If fax data is contained, it is 
regarded as Tax receiving" and the processing pro- 
ceeds to step S42. At step S42, the fax transmit- 
ting/receiving unit 7 is started and requested to perform 
processing concerning the fax receiving. The fax trans- 
mitting/receiving unit 7 stores the received fax by using 
Hs destination as a key, and informs the destination user 
the reception of the fax as occasion demands. 

At step S35, an application specified by the applica- 
tion identification information set in the received packet 
is started. At step S36, it is determined whether the 
started application contains image recognition process- 
ing. K the application does not contain image recogni- 
tion processing, processing of that application is 
executed at step S43. If the application contains image 
recognition processing, image data contained in the 
received packet is expanded at step S37 and pattern 
recognition processing and character recognition 
processing are executed on the image data at step S38. 
By this image recognition processing, the service 
requested by the user is recognized. The recognition 
processing of step S38 will be described later. 

At step S51, it is determined whether the service 
requested by the user is "mail transmission." If the serv- 
ice is "mail transmission," the processing proceeds to 
step S52. At step S52, pattern recognition processing 
and character recognition processing are executed on 
the image data expanded at step S37. While the char- 
acter recognition processing is executed for only a 
region required to recognize the service requested by 
the user at step S38, the character recognition process- 
ing is executed for the entire region of the expanded 
image data at step S52. Characters obtained by this 
character recognition processing are converted to a 
predetermined character code (such as ASCII code or 
JIS code). The character code obtained here is mail 
sentences input by the user by using the portable termi- 
nal 1 . At step S53, the mail transmitting/receiving unit 6 
is started, and the mail transmitting/receiving unit 6 
transmits the mail sentences obtained at step S52. 
Steps S52 and S53 will be described in detail later. 

At step S54, it is determined whether the service 
requested by the user is lax transmission." If the serv- 
ice is "fax transmission " the processing proceeds to 
step S55. At step S55. the fax transmitting/receiving unit 
7 is started, and the fax transmitting/receiving unit 7 
transmits the image data expanded at step S37 to a 
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transmission destination specified in the image. 

At step S56. rt is determined whether the service 
requested by the user is "mail readout." tf the service is 
"mail readout" the processing proceeds to step S57. At 

s step S57, the mail transmitting/receiving unit 6 is 
started. If mafl having the terminal ID detected at step 
S32 as its destination is received, the mafl transmit- 
ting/receiving unit 6 transmits the mail to the terminal 
having the terminal ID. Step SS7 will be described later. 

10 At step S58, ft is determined whether the service 
requested by the user is "tax takeout." If the service is 
lax takeout" the processing proceeds to step S59. At 
step S59, the fax transmitting/receiving unit 7 is started. 
If fax having the terminal ID detected at step S32 as its 

is destination is received, the lax transmitting/receiving 
unit 7 transmits the fax to the terminal having the termi- 
nal ID. 

ff the service is judged at step S74 to be recognition 
service, then the image data stored in the received 
20 packet is expanded at step S76, and the pattern recog- 
nition processing and the character recognition 
processing are executed on the image data at step S76. 
The image data subjected to the recognition processing 
is thejmage data extracted at step S11 of FIG. 4. In 
25 other words, in the example of FIG. 9 or 13, the image 
recognition processing is executed on the image data of 
the region containing only "Meeting with Mr. K on 
August 18." As a result of this image recognition 
processing, therefore, only the content written by the 
30 user as the memo are recognized. 

tf the result of recognition at step S76 contains 
characters, the characters are converted to a character 
code (such as ASCII code or JIS code) at step S77. At 
step S78, a packet for returning the recognition result to 
35 the portable terminal 1 is created. In other words, the 
transmission destination address of this packet is the 
portable terminal 1, and the above described recogni- 
tion result is stored in its data portion. At step S79, the 
created packet is sent out to the network 3. 
40 The packet sent out to the network 3 at step S79 is 
transmitted to the portable terminal 1. Upon receiving 
this packet, the portable terminal 1 displays the result of 
the image recognition processing executed by the host 
device 6 on the LCD display unit 11. Furthermore, the 
45 recognition result is stored within the portable terminal 1 
based on the user's command. 

As described above, this result of the image recog- 
nition may be stored in the host device 4. In this case, 
the recognition result is stored so as to be associated 
so with the terminal ID of the terminal device which sent 
out the difference image data (i.e.. the portable terminal 
1). Furthermore, this recognition result may be transmit- 
ted to a predetermined terminal device specified before- 
hand. In addition, retrieval processing may be executed 
55 by using a character string contained in this recognition 
result as a keyword, or the character string may be 
translated to a predetermined language. In this case, 
the retrieval result or the translation result is sent to the 
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portable terminal 1. 

There is a strong demand for reduction of portable 
terminals in size, weight and cost. In the application of 
the present embodiment, these demands can be met by 
shifting a part of the processing executed by the porta- 
ble terrnnal to the host device. For example, it is possi- 
ble to cause the host device 4 to execute the processing 
of step S1 1 of the flow chart of FIG. 4. As described 
above, step S1 1 contains the image processing. If it is 
attempted to implement such processing with high pre- 
cision or in a short time, a processor having high 
processing capability and a large size program are in 
general needed. Therefore, it is effective in reducing the 
size, weight and cost of the portable terminal 1 to cause 
the host device 4 to execute the processing of step S1 1 . 

In the case whet e the host device 4 is made to exe- 
cute the processing of step S1 1 , the portable terminal 1 
sends the image data read by using the camera 12 to 
the host device 4 as it is. In the flow chart of FIG. 1 7, the 
host device 4 executes the processing of step S1 1 after 
the processing of step S75. 

As described above, the image processing system 
of the present embodiment is so structured that an 
image of paper having a memo written thereon is taken 
in by a camera or the like of the portable terminal 1 and 
the host device 4 executes the recognition processing 
for only a required region in that image. By providing the 
portable terminal 1 with a function of taking in the image 
and transmitting it to the host device 4 and a function of 
requesting the host device 4 to execute processing (and 
in some configuration, a function of extracting a required 
region), therefore, the content of the written memo can 
be converted to an electronic ffle with such a simple 
interface that a handwritten memo or the like is input by 
using a camera. 

FIG. 18 is a detailed fbw chart of step S38 of FIG. 
16. By executing the pattern recognition processing and 
the character recognition processing on the image data 
stored in the received packet, the service requested by 
the user is detected. 

At step S61 , a specific symbol or color is detected 
from the received image data in accordance with a pre- 
determined rule. For example, in the case where brack- 
ets are used as symbols indicating the positions of a 
character string specifying the service or a character 
string specifying the transmission destination as shown 
in FIG. 5A, the image inside the brackets is extracted. In 
the case where the service classification is associated 
with a color and the transmission destination is speci- 
fied by using a character string represented by a spe- 
cific color as shown in FIG. 5B, a region using the 
specific color is extracted. 

At step S62, character recognition processing is 
executed on the image data extracted at step S61. At 
step S63. the service requested by the user is detected 
from the result of the character recognition processing. 
In the example shown in FIG. 5A or 5B, the address of 
the transmission destination of e-mail is detected simul- 



taneously with detection that the service requested by 
the user is "e-mail transmission." 

FIG. 19 is a detailed flow chart of step S52 of FIG. 
17, and shows the processing for detecting each char- 
s acter code of the character string to be transmitted as 
the mail sentences. 

At step S71, character recognition processing is 
executed on the received image data, and characters 
imaged by the portable terminal 1 are recognized. Here, 
the character recognition processing is not executed for 
the region processed at step S38. 

At step S72, the position of each of the characters 
detected at step S71 is detected. For example, the 
center position of each character is detected as a rela- 
ys tive position with respect to the top left-hand corner of 
the screen imaged by using the camera. 

At step S73, the pitch between characters is 
checked, and it is determined whether the sentences 
are written vertically or laterally. In the case of lateral 
so writing, the pitch between characters in the lateral direc- 
tion is in general smaller than the pitch between charac- 
ters in the vertical direction. In the case of vertical 
writing, the pitch between characters in the vertical 
direction is in general smaller than the pitch between 
25 characters in the lateral direction. Here, this property is 
utilized. 

At step S74, characters are extracted in the direc- 
tion obtained at step S73. For example, in the case of 
sentences written laterally, characters on a top line are 
30 extracted from the left to the right one by one. Thereaf- 
ter, the same processing is executed for the lower lines 
in order. These extracted characters are converted to a 
character code and stored in order character by charac- 
ter. 

35 By the above described processing, characters of 
sentences imaged by using the portable terminal 1 by 
the user are extracted in order and converted to charac- 
ter codes. 

FIG. 20 is a detailed flow chart of step S53 of FIG. 

40 17, and shows the mail transmission processing. This 
processing is executed by the mail transmitting/receiv- 
ing unit 6 of FIG. 1. 

At step S81, the transmission destination address 
obtained at step S38 is confirmed by the user of the 

45 portable terminal 1 In other words, the transmission 
destination address obtained by the image recognition 
processing at step S38 is transmitted to the terminal 
having the terminal ID detected at step S32, and an 
inquiry is made whether the transmission destination 

so address is correct. Upon receiving this inquiry, the port- 
able terminal 1 displays the received transmission des- 
tination address on the display unit, and waits for a 
user's command. 

At steps S82 and S83, a reply from the portable ter- 

55 minal 1 is waited for. tf there is no reply in a predeter- 
mined time, the processing ends. On the other hand, if 
there is a reply, it is determined at steps S84 and S85 
whether the reply is "OK: transmission destination 
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address conf irmed." In the case of "OK," the processing 
proceeds to step S91 . If an alteration command is con- 
tained, thefc the address is altered in accordance with 
that command at step S86, and the processing returns 
to step S81. If the reply from the portable terminal 1 is 
neither "OK" nor an alteration command, error process- 
ing is executed at step S87. 

At step S91, a TCP segment shown in FIGL 21 is 
created. In other words, a port number, a sequence 
nunte, and the like are set in a TCP header The port 
number is, for example, a number specifying an SMTP. 
As for data, the character string obtained at step S52 is 
stored as mail sentences. 

At step S92, an IP address corresponding to the 
transmission destination address included in the image 
data is acquired. The IP address is acquired by, for 
example, inquiring of a DNS (domain name server) 
which is present on the network and which is not illus- 
trated. At step S93, an IP datagram shown in FIG. 21 is 
generated. In a header of the IP datagram generated 
here, the IP address acquired at step S92 is set A data 
portion of the IP datagram has a TCP segment stored 
therein. 

At step S94, a transmission packet shown in FIG. 
21 is generated. A data portion of the transmission 
packet has the above described IP packet stored 
therein. The format of this Packet is determined by a 
communication protocol on the network 3 to which the 
host device 4 is connected. At step S95, the transmis- 
sion packet created at step S94 is sent out to the net- 
work 3. 

At step S96, it is determined whether the above 
described transmission packet could be sent out nor- 
mally. When the transmission packet could be sent out 
normally, the portable terminal 1 is informed at step S97 
that the mail transmission is conpleted and the trans- 
mitted mail is stored in the host device 4 at step S98. On 
the other hand, when the above described transmission 
packet could not be sent out normally, the portable ter- 
minal 1 is informed of that fact 

FIG. 22 is a detailed flow chart of step S57 of FIG. 
1 7, and shows the processing of reading out mail stored 
in the host device 4 from the portable terminal 1. This 
processing is executed in, for example, the following sit- 
uation. Mail directed to the user of the portable terminal 
1 is received by the mail transmitting/receiving unit 6. 
The mail transmitting/receiving unit 6 administers the 
mail. In this situation, the portable terminal 1 images the 
memo as shown in FIG. 7 by using the camera 12 and 
transmits the image data to the host device 4. 

At step S101 , the mail box of the terminal (user) of 
the inquiry source is accessed. This terminal is identi- 
fied by the terminal ID detected at the above described 
step S32. By the way. mail arrived at the mail transmit- 
ting/receiving unit 6 is put in a specific mail box accord- 
ing to its destination. For example, if mail directed to the 
user of the portable terminal 1 has arrived, the mail 
transmitting/receiving unit 6 stores it in the mail box 



installed for the user of the portable terminal 1. By 
access using the terminal ID detected at step S32 as a 
key, therefore, the letter directed to that terminal (user) 
can be read out 
5 At step S102, it is determined whether there is 
specification of a mail sending person by executing the 
character recognition processing on the received image 
data. If there is no specification, all mail stored in the 
mail box of that terminal is read out at step S103. On the 
io other hand, if there is specification of a mail sending 
person, only the mail from that mail sending person is 
read out at step S1 04. In the case of the example shown 
in FIG. 7, only the mail sent from Mr. K is read out At 
step S105, the mail read out is sent to the terminal of 
15 the inquiry source. 

In the above described embodiment, a certain 
address is specified as the transmission destination of 
an e-mail. Alternatively, a group including a plurality of 
terminals may be specified. In this case, it is necessary 
20 to register a table as shown in FIG. 23 in the mail trans- 
mitting/receiving unit 6 beforehand. It is now assumed 
that the table of FIG. 23 is set and the transmission des- 
tination address of FIG. 5A is replaced by "Business 
department." Such a memo is input by the camera 12 
25 and its image data is transmitted to the host device 4. 
By executing the character recognition processing on 
the received image data, the host device 4 recognizes 
that the mail transmission destination is the "business 
department." If at this time the table of FIG. 23 is 
30 accessed by using the terminal ID of the portable termi- 
nal 1, one or more e-mail addresses belonging to the 
•business department" are obtained. The ensuing 
processing is the same as the foregoing description. 
In the above described embodiment, the process- 
35 ing programs executed by the host device 4, i.e., the 
programs implementing the functions shown in the flow 
charts of FIGS. 1 6 through 20, and FIG. 22, and the pro- 
gram for interpreting and processing the information 
transmitted via the network are stored beforehand in the 
40 storage device 51 or the detachable recording medium 
53 in a form of a program code which can be read by the 
CPU 55. Or programs stored in another device con- 
nected via the network are used. 

FIGS. 19 through 23 show operation for an e-mail 
45 transmission. Basically, the operation for a fax transmis- 
sion is also the same. When transmitting a fax, however, 
it is necessary to detect the telephone number of the 
transmission destination by execiiBhg the image recog- 
nition processing on the received image data. For the 
so content to be transmitted as fax (fax sentences), how- 
ever, it is not necessary to execute the recognition 
processing. 

In the above described embodiment, such a config- 
uration as to take in image data by using a camera is 
55 shown. However, the present invention is not limited to 
this configuration. For example, image data read by 
using an image scanner may be transmitted to the host 
device. 
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Thus, in the image data processing system accord- 
ing to the present embodiment image data taken in by 
the portable terminal 1 is transmitted to the host device 
4. By analyzing the image data, the host device 4 recog- 
nizes the service requested by the user of the portable s 
terminal 1 . and transmits the transfer data contained in 
the image data as an e-mail or a fax according to the 
destination information contained in the image data. 
Therefore, by providing the portable terminal 1 only with 
the function of taking in image data or the like and trans- w 
mitting it to the host device 4 and the function of 
requesting the host device 4 to execute processing, an 
e-mail or the like can be transmitted with such user 
interlace that a handwritten memo or the like is imaged 
by a camera. In other words, the portable terminal 1 15 
need no execute high precision pattern recognition 
processing and various kinds of processing requiring 
high speed processing. Therefore, the portable terminal 
1 can enjoy results of sophisticated image processing 
executed in the host device 4 without having a memory 20 
of large capacity or a processor of high performance. 

Claims 

1 . A host device for a system connecting various port- 25 
able terminals via a network, characterized by com- 
prising: 

means (6, 7) for receiving data including image 
data transmitted from a portable terminal and 30 
executing character recognition on specific 
information included in said image data; and 
means (5) for executing a command indicated 
by a character string which is recognized by 
said character recognition, with respect to the 35 
received data, and transmitting received data 
to a given portable terminal. 

2. A host device according to claim 1 , characterized in 
that in said received image data, information indi- 40 
eating the transmission destination is included so 

as to be discriminated by identification information, 
a character string discriminated by said identifica- 
tion information is recognized as information of 
transmission destination, and data is transmitted to 45 
the transmission destination. 

3. A host device according to claim 1 , characterized in 
that said specific information included in said image 
data is a command indicating an e-mail transmis- so 
sion, and processing of e-mail transmission is per- 
formed based on the recognized command. 

4. A host device according to claim 1 , characterized in 
that said specific information included in said image ss 
data is a command indicating a fax transmission, 
and processing of fax transmission is performed 
based on the recognized command. 
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5. A host device according to claim 1 , characterized in 
that said specific information included in said image 
data is a command indicating an inquiry of received 
mail, and mail check is performed on the basis of 
the recognized command. 

6. A portable terminal device characterized by com- 
prising: 

means (1a) for reading, as an image, data indi- 
cating transmission destination and transmis- 
sion data so as to be distinguishable from each 
other; and 

means (1b) for transmitting the image data thus 
read and the command indicating the process- 
ing instruction for said image data to a host 
device connected to a network. 

7. A portable terminal device according to claim 6, 
characterized in that said command indicating the 
processing instruction is a command specifying 
mail transmission or fax transmission. 



EP 0865 192 A2 




EP 0 865 192 A2 




FIG.2 



EP0 865192A2 



CO. 
CM 



o 
O 






















o 







> 

cc 
o 
s 

Ui 

s 




CO 

(5 
LL 



CM 



to 

CM 







3o 


ORDING 
M DRIVE 






DETACHAB 
RECORDIN 
MEDIUM 


REC 
EDIU 




2 







EP 0 865 192 A2 



( WAIT INPl/T) 




S13 

!! L_ 

EXECUTE 
OTHER 
PROCESSING 



REWRITE IMAGE DATA 
ACCORDING TO PEN INPUT 



S4 



X 



DISPLAY USER S _ C c 
COMMAND INPUT SCREEN &a 



USER'S 
COMMAND 



RECOGNIZE 
USER'S COMMAND 




S13 



DISCARD 
IMAGE DATA 





CREATE PACKET 






SENC 


>OUT 



S8 



STORE IMAGE 
ON SIDE OF 
PORTABLE 
TERMINAL 



~S9 



( END ) 



FIG.4 



EP 0 8651 92 A2 



[E-mail transmission] = [abc@xyz.co.jp] 

Hello! Thank you for your e-mail. As for 
the schedule of the next week, • • • 



i 



FIG.5A 



abc@xyz.co.jp 

Hello! Thank you for your e-mail. As for 
the schedule of the next week, • • • 



FIG.5B 



EP 0 865 192 A2 



[Fax transmission] = [03 - 3XXX - A AAA] 



+-H4 



I I l I I I I I I l I I 



J I 



QS 



FIG.6A 



MAIL TRANSMISSION FORMAT 



TRANSMISSION 
DESTINATION 
ADDRESS 



Hello! Thank you for your e-mail. 
As for the schedule of the 
next week, ■ • • 



ADDRESS 
ENTERING 
REGION 

MAIL 
SENTENCE 
ENTERING 
REGION 



FIG.6B 



EP0 865192A2 



[Mail inquiry] 



from Mr.k 



EP 0 865 192 A2 



[E-mail transmission] = [abc@xyz.co.jp] 

A portable terminal of XXXX put on the 
market this month has enriched mail 

functions, 

By the way 




FIG.8A 



[E-mail transmission] = [abc@xyz.co.jp] 

A portable terminal of XXXX put on the 
market this month has enriched mail 

functions, 

By ttni way 



FIG.8B 



EP0 865192 A2 



PAPER 



1 



ABCDEFGHIJKLM 


nop: 


QRSTUVWXYZ 








Meeting with Mr.k on J 
August 18 / 








f } 



FIG.9 



EP0 865 192 A2 



1 



HEADER 



DATA 



TRANSMISSION 
SOURCE ADDRESS 
DESTINATION 
ADDRESS 



APPUCATION IDENTIFICATION 
INFORMATION (COMMAND) 
TERMINAL ID 
IMAGE DATA 
TIME INFORMATION 



FIG.10A 



( START ) 

JL 



COMPRESS IMAGE DATA AND STORE 
COMPRESSED IMAGE DATA 



SET TIME INFORMATION 



S21 
[~S22 



3. 



SET APPLICATION IDENTIFICATION 



I 



~S23 



SET TERMINAL ID 



I 



l- 



S24 



CREATE HEADER (SUCH AS TRANSMISSION 

SOURCE ADDRESS AND DESTINATION S25 
ADDRESS SETTING) 

♦ " 

( END ) 



FIG. 1 0B 



EP0865192A2 



( START ) 

jr 

EXECUTE LINE DETECTION i 
PROCESSING ON INPUT IMAGE DATA ~~ b31 



EXECUTE LINE CONNECTION 
PROCESSING 



-S32 



DETECT CLOSED LOOP C<M 
(CLOSED RESION) ~5>J3 
GREATER THAN PREDETERMINED AREA 



I 



OUTPUT IMAGE DATA WITHIN 
CLOSED REGION 



( END ) 



S34 



FIG.1 1 



EP 0 865 192 A2 




FIG.12 



EP0 865 192 A2 



, QRSTUVWXYZ 

i Meeting with Mr.k 

[ on August 18 

k__________ 

FIG.13A 

! QRSTUVWXYZ 

• Meeting with Mr.k 

| on August 18 

FIG.13B 



EP 0 865 192 A2 



( START ) 



EXECUTE UNE DETECTION ^ a L-S41 
PROCESSING ON INPUT IMAGE DATA p ^ 



EXECUTE LINE CONNECTION Ls42 
PROCESSING 

I 



DETECT LINES LONGER THAN 
PREDETERMINED LENGTH IN _ $43 
HORIZONTAL DIRECTION OR NEARLY ^ 
HORIZONTAL DIRECTION 



I 



ALONG DETECTED LINE, CUT OUT 
REGION ABOVE LINE WITH _ 
PREDETERMINED WIDTH AND OUTPUT 
IMAGE DATA WITHIN THAT REGION 



~S44 



7t 

C END ) 



FIG.14A 



( START ) 



CUT OUT REGION CONTAINING PIXELS 
OF PREDETERMINED COLOR FROM S51 
INPUT IMAGE DATA, AND OUTPUT "~ 
IMAGE DATA CONTAINED IN THAT REGION 

( END ) 



FIG.14B 



EP 0 865 192 A2 



O 



S3 



NETWORK 

t i 



52- 



RECORDING 
MEDIUM 
DRIVER 



COMMUNICATION 
CONTROLLER 



-54 




MEMORY 



T 

56 



STORAGE 
DEVICE 



I 

51 



FIG.15 



EP0 865192A2 




S35 



START APPLICATION SPECIFIED 
BY APPLICATION SPECIFYING 
INFORMATION 



1 



S42 

1 



1 



S41 

_L_ 



EXECUTE MAIL 

RECEIVING 
PROCESSING 



EXECUTE FAX 

RECEIVING 
PROCESSING 



S36 



application 
containing image 
recognition 
processing 

'yes 



S43 

? 



EXECUTE 
OTHER 
PROCESSING 



EXPAND 
IMAGE DATA 



~S37 



EXECUTE PATTERN 
RECOGNITION 
(CHARACTER 
RECOGNITION) 



5 



~S38 



( END ) 



FIG.16 



EP0 865192 A2 



S55 

_L_ 



YES 



EXECUTE FAX 
TRANSMISSION 
PROCESSING 



S59 

1 



YES 



EXECUTE FAX 

TAKEOUT 
PROCESSING 




1 



S52 

__L_ 



CONVERT 
RECOGNIZED 
CHARACTERS TO 
CHARACTER CODE 



X 



EXECUTE MAIL 
TRANSMISSION 
PROCESSING 



1 



EXECUTE MAIL 

READOUT 
PROCESSING 



S57 



EXECUTE PATTERN RECOGNITION 
(CHARACTER RECOGNITION) 



T 



]~S76 



CONVERT RECOGNIZED 
CHARACTERS TO CHARACTER CODE 



T 



~S77 



CREATE PACKET 



I 



~S78 



SEND OUT 
(SEND OUT RECOGNITION RESULT 
TO TERMINAL WHICH TRANSFERRED 
IMAGE DATA) 



~S79~ 



( END ) 



FIG.17 



EP 0 865 192 A2 



( START ) 



DETECT SPECIFIC SYMBOLS _c 61 
OR COLOR r^wi 



EXECUTE CHARACTER 
RECOGNITION PROCESSING 
ON DETECTED IMAGE DATA 



— S62 



DETECT USER'S REQUEST 



S63 



( END ) 



FIG.18 



EP0 865192A2 



( START ) 



EXECUTE CHARACTER 
RECOGNITION PROCESSING 
ON IMAGE DATA AND 
DETECT CHARACTERS 



DETECT POSITION OF 
EACH CHARACTER. 



— S71 



— S72 



CHECK PITCH BETWEEN 

CHARACTERS AND 
DETERMINE WHETHER 
SENTENCES ARE WRITTEN 
VERTICALLY OR LATERALLY 



— S73 



EXTRACT CHARACTERS UNE 
BY LINE AND CONVERT 
CHARACTERS TO 
CHARACTER CODE 



S74 



( END ) 



FIG.19 



EP 0 865 192 A2 




ACQUIRE IP ADDRESS 
CORRESPONDING TO 

TRANSMISSION 
ADDRESS ENTERD IN 
IMAGE DATA 



ALTER ADDRESS 
AS PER COMMAND 



EXECUTE 
ERROR 
PROCESSING 



( END ) 



S87 



CREATE TCP DATAGRAM 



I 



CREATE 
COMMUNICATION PACKET 




S93 
S94 
S95 



INFORM OF 
TRANSMISSION 
COMPLETION 



T 



~S97 



INFORM OF m 
TRANSMISSION h~S99 
FAILURE 



STORE MAIL SENTENCES ~S98 



( END ) 



FIG.20 



EP0 865192A2 



PORT NUMBER 
SEQUENCE NUMBER 



TCP HEADER 


DATA 


X. i 
ADDRESS X 


IP HEADER 


DATA 



-•-TCP SEGMENT 



IP DATAGRAM 



HEADER 



DATA 



TRAILER 



TRANSMISSION 
PACKET 



FIG.21 



EP0 865192A2 



C START ) 



ACCESS MAIL BOX 
OF INQUIRY SOURCE 



— S101 




S103 



READ OUT ALL MAIL 


\ 




f 


TRANSIT 
READ< 
INQUIRY 


HIT MAIL 
JUT TO 
SOURCE 



r 



S104 



READ OUT MAIL 
RECEIVED FROM 

SPECIFIED 
SENDING PERSON 



— S105 



( END ) 



FIG.22 



EP 0 865 192 A2 



GROUP NAME 


E-MAIL ADDRESS 


BUSINESS 
DEPARTMENT 


abc@xyz.co.jp 
def@xyz.co.jp 

• 
■ 


SAME YEAR 
CLUB 


ghi@xxx.co.jp 
jkl@yyy.or.jp 

■ 

* 
• 


• 
■ 
• 





FIG.23 



(19) 



J 



Europiisches Patentamt 
European Patent Office 
Office europeen des brevets 




(12) 



(88) Date of publication A3: 

17.0U000 Bulletin 2000/20 

(43) Date of publication A2: 

16.09.1998 Bulletin 1998/38 

(21) Application number: 98104305.2 

(22) Dateof filing: 10.03.1998 



(id EP 0 865 192 A3 

EUROPEAN PATENT APPLICATION 

(51) Int. CI. 7 : H04N1/32 



(84) 


Designated Contracting States: 


(72) 


Inventor: 




AT BE CH DE DK ES Fl FR GB GR IE IT LI LU MC 




Yamakita, Tooru, 




NL PT SE 




c/o Casio Computer Co., Ltd. 




Designated Extension States: 




Hamura-shi, Tokyo 205-8555 (JP) 




AL LT LV MK RO SI 










(74) 


Representative: 


(30) 


Priority: 11.03.1997 JP 5578197 




Grunecker, Kinkeldey, 


19.03.1997 JP 6574697 




Stockmair & SchwanhSusser 








AnwaltssozietSt 


(71) 


Applicant: 




Maximilianstrasse 58 




Casio Computer Co., Ltd. 




80538 Munchen(DE) 




Shibuya-ku, Tokyo (JP) 







CO 

< 

CM 

o> 

lO 
CO 
00 

o 

Q. 



(54) Portable terminal device for transmitting image data via network and image processing 

device for performing an image processing based on recognition result of received image 
data 



(57) A user of a portable terminal (1) writes sen- 
tences to be transmitted as an e-mail, a mail address, 
and information indicating that service a host device (4) 
is requested to provide is "mail transmission" on paper 
as a memo. By using an image input unit (1a) installed 
in the portable terminal, the paper or the like is imaged. 
The portable terminal transmits the image data thus 
taken in to the host device. The host device analyzes 
received image data by using an image recognition unit 



(5). Upon recognizing by characters that the service 
requested by the portable terminal is "mail transmis- 
sion", the host device starts a mail transmitting/receiv- 
ing unit (6). The mail transmitting/receiving unit (6) 
transmits the content of the written memo included in 
the image data to a terminal specified by the mail 
address included in the image data. 




FIG.1 




EP 0 865 192 A3 



European Patent 
Office 



EUROPEAN SEARCH REPORT 



Application Number 

EP 98 10 4305 



DOCUMENTS CONSIDERED TO BE RELEVANT 



Category 



Citation of document with indication, where appropriate, 
of relevant passages 



Relevant 
to claim 



CLASSIFICATION OF THE 
APPLICATION (tntCI.6) 



US 5 287 199 A (Z0CC0LILL0 SUSAN H) 
15 February 1994 (1994-02-15) 
the whole document * 

PATENT ABSTRACTS OF JAPAN 

vol. 1997, no. 04, 

39 April 1997 (1997-04-30) 

& JP 08 331286 A (TOSHIBA CORP), 

13 December 1996 (1996-12-13) 

* abstract * 

MOORE L C: "ELECTRONIC ROUTING OF 
FACSIMILE IMAGES" 
XEROX DISCLOSURE JOURNAL, US, XEROX 
CORPORATION. STAMFORD, CONN, 
vol. 18, no. 1, 

1 January 1993 (1993-01-01), page 73 
XP0OO333950 

* the whole document * 

EP 6 465 Oil A (HEWLETT PACKARD CO) 
8 January 1992 (1992-01-08) 

* abstract * 



1,2,4,6 



1,2,4,6 



H04N1/32 



1,2,4,6 



The present search report has been drawn up for all claims 



1-3,5-7 



TECHNICAL FIELDS 
SEARCHED (lnt.CI.6) 



H04N 



Place of search 

THE HAGUE 



Date of completion of the search 

27 March 2000 



Examiner 

Hazel, J 



CATEGORY OF CITED DOCUMENTS 

X : particularly relevant if taken alone 

Y : partioutarty relevant if combined with another 

documentor the same category 
A : technological background 
O : non- written eSsotosure 
P : intei i mn. In if u document 



T : theory or principle underlying the invention 
£ : earlier patent document, but pub fished on, c 

after the filing date 
D : document cited in the application 
L : document cited for other reasons 



& : member of the same patent family, corresponding 
document 



EP 0 865 192 A3 



ANNEX TO THE EUROPEAN SEARCH REPORT 

ON EUROPEAN PATENT APPLICATION NO. EP 98 10 4305 



This annex lists the patent family members relating to the patent documents cited in the above-mentioned European search report. 
The members are as contained in the European Patent Office EDP file on 

The European Patent Office is in no way liable for these particulars which are merely given for the purpose of information. 

27-03-2GOO 



Patent document 
cited in search report 


Publication 
date 


Patent family 
members) 


Publication 
date 


US 5287199 


A 


15-02-1994 


CA 2089195 A,C 


28-08-1993 


JP 08331286 


A 


13-12-1996 


NONE 




EP 0465011 


A 


08-01-1992 


US 5115326 A 
DE 69122991 D 
DE 69122991 T 
JP 5199388 A 


19-05-1992 
12-12-1996 
06-03-1997 
06-08-1993 



i 



& For more details about this annex : see Official Journal of the European Patent Office, No. 12/82 



THIS PAGE BLANK (usfto) 



•Patent Abstracts of Japa 



PUBLICATION NUMBER 
PUBLICATION DATE 



t 



tUHUKtAN KAItlMI 



APPLICATION DATE 
APPLICATION NUMBER 

APPLICANT : RICOH CO LTD; 

INVENTOR : KATSUMATA MAIKO; 

INT.CL. : H04N 1/46 



06030288 
04-02-94 

13-07-92 
04185023 



as-**** 



!2 



16 



17 



M cpu "^ .is 



ROM 



.19 



— RAM _20 



TITLE 



COLOR FACSIMILE EQUIPMENT 



21 — ^ 



22-^ 



! I© © ©i 
!© © ® 
»® © ® 



.24 



23 
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